%% BioMed_Central_Tex_Template_v1.05
%% %
% bmc_article.tex ver: 1.05 %
% %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% LaTeX template for BioMed Central %%
%% journal article submissions %%
%% %%
%% <27 January 2006> %%
%% %%
%% %%
%% Uses: %%
%% cite.sty, url.sty, bmc_article.cls %%
%% ifthen.sty. multicol.sty %%
%% %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %% 
%% For instructions on how to fill out this Tex template %%
%% document please refer to Readme.pdf and the instructions for %%
%% authors page on the biomed central website %%
%% http://www.biomedcentral.com/info/authors/ %%
%% %%
%% Please do not use \input{...} to include other tex files. %%
%% Submit your LaTeX manuscript as one .tex document. %%
%% %%
%% All additional figures and files should be attached %%
%% separately and not embedded in the \TeX\ document itself. %%
%% %%
%% BioMed Central currently use the MikTex distribution of %%
%% TeX for Windows) of TeX and LaTeX. This is available from %%
%% http://www.miktex.org %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\NeedsTeXFormat{LaTeX2e}[1995/12/01]
\documentclass[10pt]{bmc_article} 
% Load packages
\usepackage{cite} % Make references as [1-4], not [1,2,3,4]
\usepackage{url} % Formatting web addresses 
\usepackage{ifthen} % Conditional 
\usepackage{multicol} %Columns
\usepackage[utf8]{inputenc} %unicode support
%\usepackage[applemac]{inputenc} %applemac support if unicode package fails
%\usepackage[latin1]{inputenc} %UNIX support if unicode package fails
\urlstyle{rm}

\usepackage{pbox}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
%% %%
%% If you wish to display your graphics for %%
%% your own use using includegraphic or %%
%% includegraphics, then comment out the %%
%% following two lines of code. %% 
%% NB: These line *must* be included when %%
%% submitting to BMC. %% 
%% All figure files must be submitted as %%
%% separate graphics through the BMC %%
%% submission process, not included in the %% 
%% submitted article. %% 
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\def\includegraphic{}
\def\includegraphics{}
% *** GRAPHICS RELATED PACKAGES ***
%
% or other class option (dvipsone, dvipdf, if not using dvips). graphicx
% will default to the driver specified in the system graphics.cfg if no
% driver is specified.
\usepackage[dvips]{graphicx}
% declare the path(s) where your graphic files are
% \graphicspath{{../eps/}}
% and their extensions so you won't have to specify these with
% every instance of \includegraphics
\DeclareGraphicsExtensions{.eps}
\setlength{\topmargin}{0.0cm}
\setlength{\textheight}{21.5cm}
\setlength{\oddsidemargin}{0cm} 
\setlength{\textwidth}{16.5cm}
\setlength{\columnsep}{0.6cm}
\newboolean{publ}

\usepackage{rotating}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% You may change the following style settings %%
%% Should you wish to format your article %%
%% in a publication style for printing out and %%
%% sharing with colleagues, but ensure that %%
%% before submitting to BMC that the style is %%
%% returned to the Review style setting. %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Review style settings
\newenvironment{bmcformat}{\begin{raggedright}\baselineskip20pt\sloppy\setboolean{publ}{false}}{\end{raggedright}\baselineskip20pt\sloppy}
%Publication style settings
%\newenvironment{bmcformat}{\fussy\setboolean{publ}{true}}{\fussy}
% Begin ...
\begin{document}
\begin{bmcformat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% Enter the title of your article here %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\title{As strong as the weakest link: Handling compromised components in OpenStack}
% New suggestion - feel free to change
\title{Handling Compromised Components in an IaaS Cloud Installation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% Enter the authors here %%
%% %%
%% Ensure \and is entered between all but %%
%% the last two authors. This will be %%
%% replaced by a comma in the final article %%
%% %%
%% Ensure there are no trailing spaces at %% 
%% the ends of the lines %% 
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\author{Aryan TaheriMonfared\correspondingauthor$^{1}$%
\email{Aryan TaheriMonfared\correspondingauthor - aryan@uninett.no}%
and
Martin Gilje Jaatun$^2$%
\email{Martin Gilje Jaatun - Martin.G.Jaatun@sintef.no}
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% Enter the authors' addresses here %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\address{%
\iid(1)UNINETT, Trondheim, Norway\\
\iid(2)SINTEF ICT, Trondheim, Norway
}%
\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% The Abstract begins here %%
%% %%
%% The Section headings here are those for %%
%% a Research article submitted to a %%
%% BMC-Series journal. %% 
%% %%
%% If your article is not of this type, %%
%% then refer to the Instructions for %%
%% authors on http://www.biomedcentral.com %%
%% and change the section headings %%
%% accordingly. %% 
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
% Do not use inserted blank lines (ie \\) until main body of text.
This article  presents an approach to handle compromised components in the OpenStack Infrastructure-as-a-Service cloud environment. We present two specific use cases; a compromised service process and the introduction of a bogus component, and we describe several approaches for containment, eradication and recovery after an incident. Our experiments show that traditional incident handling procedures are applicable for cloud computing, but need some modification to function optimally.
\end{abstract}
\ifthenelse{\boolean{publ}}{\begin{multicols}{2}}{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% The Main Body begins here %%
%% %%
%% The Section headings here are those for %%
%% a Research article submitted to a %%
%% BMC-Series journal. %% 
%% %%
%% If your article is not of this type, %%
%% then refer to the instructions for %%
%% authors on: %%
%% http://www.biomedcentral.com/info/authors%%
%% and change the section headings %%
%% accordingly. %% 
%% %%
%% See the Results and Discussion section %%
%% for details on how to create sub-sections%%
%% %%
%% use \cite{...} to cite references %%
%% \cite{koon} and %%
%% \cite{oreg,khar,zvai,xjon,schn,pond} %%
%% \nocite{smith,marg,hunn,advi,koha,mouse}%%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% %%
%% Figures %%
%% %%
%% NB: this is for captions and %%
%% Titles. All graphics must be %%
%% submitted separately and NOT %%
%% included in the Tex document %%
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%\textbf{TODO: Express the difference in section 2 and 3, i.e. they are not repeating the same concept}
Although Cloud Computing has been heralded as a new computing model,
it is fundamentally an old idea of providing computing resources as a
utility\cite{utility}. This computing model will reduce the upfront
cost for developing and deploying new services in the
Internet. Moreover, it can provide efficient services for special
use-cases which require on-demand access to scalable resources.

Cloud Computing has a variety of service models and deployment models
which have been in use in various combinations for some time \cite{nist:cloud_definition}. The chosen
service and deployment model of a cloud environment will determine
what kind of vulnerabilities might threaten it.
One of the main obstacles in the movement toward Cloud Computing is
the perceived insufficiency of Cloud security. Although it has been
argued \cite{Chen:EECS-2010-5} that most of the security issues in
Cloud Computing are not fundamentally novel, a new computing model
invariably brings its own security doubts and issues to the market.

In a distributed environment with several stakeholders, there will
always be numerous ways of attacking and compromising a component, and
it is not possible to stop all attacks or ensure that the system is
secure against all threats. Thus, instead of studying attack methods, a better approach is to
assess the risk and try to understand the impact of a compromised
component. 
%In order to study impacts of a successful attack, 
To do this, the exact
functionalities of each component
% are extracted.  
must be determined, after which  
%After identifying impacts of a successful attack, we should find 
efficient approaches to
tolerate such an attack can be identified. 
% and its damages. 
%In this process, the incident
%should be detected and analyzed first. Detecting and analyzing an
The first step of this process is to detect, and then analyze the incident, something which is subject to 
%incident have 
a set of best practice procedures
%. These procedures 
which are
dependent on 
%the 
knowledge about the normal behavior and operation of
the system. The next step is about containing the incident.  There are
currently several public cloud providers; however, none of them
disclose their security mechanisms. 
%Thus, we should 
This highlights the need to study applicable
mechanisms and introduce new ones to fulfill security requirements of
a given cloud environment; in this article, we describe our work on an
open-source deployment of a cloud environment based on the OpenStack
cloud platform. 
%By publishing these approaches, we hope to enable
%other researchers to also analyze them and make them more
%robust. Applying the same steps and best practices to a different set of
%platform applications (e.g. OpenNebula) can result in useful
%information about effectiveness, efficiency, robustness, and
%appropriateness of the introduced mechanisms.
\begin{figure}
\centering
\includegraphics[scale=0.4]{figures/LabDetail}
\caption{Lab setup}
\label{figure:LabDetail} 
\end{figure}
%\section{Components at Risk}

When we talk about a compromised component in this document, we mean those components in a cloud environment that are disclosed (i.e., private contents revealed), modified, destroyed or even lost \cite{aryan-cloudcom}. Finding compromised components and identifying their impacts on a cloud environment is crucial.
%% It will help stakeholders plan appropriate incident handling
%% strategies for their cloud environment in case of facing a compromised
%% component.
%\textit{\textbf{TODO}: ... we will use OpenStack as the cloud software for our study \& we will focus on the OpenStack Compute project (Nova)...}

\subsection{A Brief Primer on OpenStack}
We have found the OpenStack cloud platform to be the best choice for a
real case study in our research. In our laboratory configuration, we
used the simple flat deployment structure. This will avoid further complexity
which 
%is 
would be caused by %the 
a hierarchical or peer-to-peer architecture. We
have four physical machines; one of them will be the cloud controller,
and other three are compute worker nodes. The abstract diagram of our
lab setup is depicted in Figure \ref{figure:LabDetail}.
It should be noted that although we focus on the OpenStack as a
specific cloud software in our study, more or less the same components and
processes can be found in other cloud platform implementations.
%openstack

OpenStack consists of a set of open-source projects which provide a
variety of services for an Infrastructure as a Service (IaaS)
model. Its five main projects deliver basic functionalities that are
required for a cloud infrastructure, comprising: Nova (compute), Swift
(storage), Glance (VM image), Keystone (identity), and Horizon
(dashboard). The OpenStack community is fairly big, with a lot of
leading companies involved. A big community for an open-source project
has its own advantages and disadvantages, but further discussion on
this topic is out of the scope of this article.

The Compute project (Nova) provides fundamental services for hosting
virtual machines in a distributed yet connected environment. It
handles provisioning and maintenance of virtual machines, as well as
exposing appropriate APIs for cloud management. The object storage project
(Swift) is responsible for delivering a scalable, redundant, and
permanent object storage. It does not facilitate a regular file system
in the cloud. Virtual machine disk images are handled by the Image Service
project (Glance). Discovery, uploading, and delivery of images are
exposed using a REST\footnote{REpresentational State Transfer} interface. The image service does not store the
actual images, but utilizes other storage services for that purpose,
such as OpenStack Object Storage. The identity project (Keystone)
unifies authentication for the deployed cloud infrastructure. Cloud
services are accessible through a portal provided by the dashboard
project (Horizon) \cite{openstack:cactus}.

The OpenStack architecture is based on a Shared Nothing (SN) and
Message Oriented architecture. Thus, most of the components can run on
multiple nodes and their internal communication functions in a
synchronous fashion via a messaging system. In this deployment (and in
the default installation of OpenStack), RabbitMQ is used as the
messaging system. RabbitMQ
% which 
is based on the Advanced Messaging Queue Protocol
(AMQP) standard. These architectures are used to avoid common
challenges in a distributed environment, such as deadlock, live lock,
etc.

We have decided to focus on the Compute project of OpenStack, which
has enabled us to dive deeply into the details, and exercise different
modules in the Compute project. However, the same results are
applicable to the rest of the OpenStack projects. All projects follow
the same architectural concepts and design patterns, so despite their
functionalities, their behavior in a distributed and highly scalable
environment would be similar.

\begin{figure}[h!]
\centering
\includegraphics[scale=.4]{figures/nova-overview2}
\caption{Nova components and their interaction\cite{openstack-wiki:ArchitecturalOverview} } 
\label{figure:nova-overview2} 
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[scale=.3]{figures/NovaComponents_Arch}
\caption{OpenStack Compute basic architecture \cite{openstack-wiki:MultiClusterZones}}
\label{figure:NovaComponents_Arch} 
\end{figure}

OpenStack Compute has 5 interacting modules, comprising: compute
controller, network controller, volume controller, scheduler, and API
server. They provide basic functionalities for hosting, provisioning
and maintaining virtual machine instances. The compute controller
interacts with the underlying hypervisor and allocates required
resources for each virtual machine. The network controller provides
networking resources (e.g. IP addresses, VLAN specification, etc.) for
VM instances. The volume controller handles block storages devices for
each VM instance, and the scheduler distributes tasks among worker
nodes (i.e. compute controllers). The API server exposes all these
functionalities to the outside world.

\subsection{Article structure}
We will continue to discuss general aspects of incident handling in a specific cloud environment, and our case studies for possible attack scenario to such a model.

The rest of the paper is structured as follows: First, we will explain
the adapted form of NIST incident handling guideline for the cloud
model (Section \ref{section:incident handling}). Then two incidents
will be processed by the adapted guideline (Section
\ref{subsection:case studies}). Applying the guideline leads us to a
set of new challenges that have not been addressed previously or
require a careful re-analysis. Finally, by analyzing these challenges,
a group of security mechanisms are proposed which address existing
deficiencies (Section \ref{section:approaches}). A brief comparison of
mechanisms are provided as well (Section \ref{subsection:comparison}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Incident handling}\label{section:incident handling}
% \textbf{ Change the title?}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\subsection{Detection and analysis of the compromised component}
%We studied different characteristics of cloud components. 
We will focus on cloud platform components, specifically on their functionalities,
access methods, interacting components and the impacts in case of being compromised. 
%% We will
%% use outcomes of components analysis to study detection methods and
%%% analyze compromised components. Digging 
%impacts of a compromised
%component will reveal its symptoms. 
The symptoms of a compromised component are useful in detecting
security breaches and must be considered when performing further
analysis.  Studying the detection and analysis phase of the incident
handling procedure, and applying new characteristics of the Cloud
Computing model, we identified several requirements for a cloud
provider and a cloud consumer. Additionally, some influential
challenges which will hinder implementation of these requirements or
adaptation of existing mechanisms will be explained.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Detection and Analysis of the compromised component}
Studying the detection and analysis phase of the NIST incident
handling guideline \cite{SP800-61Rev.1}, and applying new
characteristics of the Cloud Computing model, we identified several
requirements for a cloud provider and a cloud consumer. 
%% Additionally,
%% some influential challenges have been explained which will hinder
%% implementation of these requirements or adaptation of existing
%% mechanisms.
\subsubsection{Cloud providers' requirements}
The cloud provider should develop the following items to play its role
in the incident handling process. Most of these items are
orthogonal. In other words, a cloud consumer may request several items
(i.e. security functionalities, services) together. Also, different
consumers may not have similar demands. Thus, it 
%is better 
may be beneficial for the provider to develop
most (if not all) of the following items if it wishes  to cover a larger set of consumers.
\begin{itemize}
\item \textbf{Security APIs:} The cloud provider should
develop a set of APIs that deliver event monitoring
functionalities and also provide forensic services for
authorities. Event monitoring APIs ease systematic incident
detection for cloud consumers and even third
parties. Forensic services at virtualization level can be
implemented by means of virtual machine introspection
libraries. An example of an introspection library is 
XenAccess that allows a privileged domain to access live
states of other virtual machines. A cross-layer security
approach seems to be the best approach in a distributed
environment \cite{TaheriMonfared:monitoring}. 
%% This approach
%% should be implemented and analyzed in a real case
%% environment to study its advantages and disadvantages.
\item \textbf{Precursor or Indication Sources:} The cloud
provider deploys, maintains and administrates the cloud
infrastructure. The provider also develops required security
sensors, logging and monitoring mechanisms to gather enough
data for incident detection and analysis at the
infrastructure level. As an example, security agents,
intrusion monitoring sensors, application log files, report
repository, firewall statistics and logs are all part of
security relevant indication sources. In case of a security
incident, the cloud provider should provide raw data from
these sources to affected customers and stakeholders. Thus
they will be capable of analyzing raw data and
characterizing incident properties.
% This approach has its own challenges which will be discussed
% in the next section.
\item \textbf{External reports:} The cloud provider should provide a
  framework to capture external incident reports. These incidents can
  be reported by cloud consumers, end users or even third
  parties. This is not a new approach in handling an incident, however
  finding the responsible stakeholders for that specific incident and
  ensuring correctness of the incident\footnote{Avoiding false
    positive alarms} requires extensive research. 
%An illustration,
 E.g.,  Amazon has developed a "Vulnerability Reporting
  Process"\cite{amazon:vulnerability-reporting} which delivers these 
%same
  functionalities. 
%as described before.
% \item \textbf{Cloud provider's responsibilities:}
\item \textbf{Stakeholder interaction:}
A timely response to an incident requires heavy interaction of stakeholders. In order to ease this interaction at the time of crisis, the responsibilities of each stakeholder should be described in detail. 
\item \textbf{Security services:} Cloud consumers may not be
  interested in developing security mechanisms. The cloud provider can
  deliver a security service to overcome this issue. Security services
  which are delivered by the provider can be more reliable in case of
  an incident and less challenging in the deployment and the incident
  detection/analysis phases.
%% When a provider delivers a security service for its customers,
%% the provider already knows about its own infrastructure; thus
%% it won't face any problems in evidence gathering or incident
%% analysis because of missing information about underlying
%% architecture or limited access to indication sources.
\item \textbf{Infrastructure information:} When the cloud
consumer or another third party wants to develop incident
detection and analysis mechanisms, they %may 
will need to
understand the underlying infrastructure and its
architecture. However, without cloud provider cooperation
that will not be feasible. So, the cloud provider should
disclose enough information to responsible players to detect
the incident in a timely fashion and study it to propose the
containment strategy. 
%\textbf{Is this one feasible at all? discuss some doubts ...}
\end{itemize}
\subsubsection{Cloud consumers' requirements}
A cloud consumer 
%, as well as its provider, has several responsibilities and 
must fulfill requirements to ensure effectiveness
of the incident detection and analysis process.
%The following contains identified requirements or possible approaches for a cloud consumer:
\begin{itemize}
\item \textbf{Consumer's security mechanisms:} The cloud
consumer might prefer to develop its own security
mechanisms (e.g. incident detection and analysis
mechanisms). The customer's security mechanisms can be based on
either the cloud provider's APIs or reports from a variety of
sources, including: provider's incident reports, end-users'
vulnerability reports, third parties' reports.
\item \textbf{Provider's agents in customer's resources:}
By implementing provider's agents, the cloud consumer will facilitate approaching a cross-layer security solution. In this method, the cloud consumer will know the exact amount and type of information that has been disclosed. Moreover, neither the cloud consumer nor the provider needs to know about each others' architecture or infrastructure design.
\item \textbf{Standard communication protocol:}
In order to have systematic incident detection and analysis mechanisms, it is required to agree on a standard communication protocol that will be used by all stakeholders. This protocol should be independent of a specific provider/customer.
\item \textbf{Report to other stakeholders:}
If the customer cannot implement the provider's agent in its own instances, another approach to informing stakeholders about an incident is by means of traditional reporting mechanisms.
These reports should not be limited to an incident only, customers may also use this mechanism to announce a suspicious behavior for more analysis.
\item \textbf{Cloud consumer's responsibilities:} Roles and
responsibilities of a cloud consumer in case of an incident
should be defined ahead of time, 
%previously,
%; thus it will be feasible to 
facilitating immediate reaction in a
crisis. 
%% It should be clear that after detecting the first
%% symptoms of an incident, the cloud consumer must start
%% communicating with which components of a cloud and expect
%% what kind of responses.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Case studies}\label{subsection:case studies}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We now present two examples that illustrate handling a compromised
node and an introduced bogus node, respectively.

\subsubsection{Case One: A Compromised Compute Worker}
%The first case which we will discuss, has only one compromised component. 
In the first case only one component, the nova-compute service in the
compute worker, is compromised, as shown in Figure
\ref{figure:LabAbstract-Case1}.  Two incidents have happened
simultaneously in this scenario: malicious code and unauthorized
access. The malicious code is injected to the nova-compute service and
introduces some misbehavior in it, such as malfunctions in the hosting
service of virtual machine instances. 
%As illustration of 
A malfunction
can be provoked, e.g., through nefarious use of granted privileges to request 
%for 
more IP
addresses, causing IP address exhaustion. The incident description
for this scenario is given in Table \ref{table:case one}.

The malicious code is injected after another incident, %like
unauthorized access. The attacker gains access to resources on the
OpenStack-4 host, that he/she was not intended to have. Using those
escalated privileges, the attacker changed the python code of the
nova-compute and restarted the service, %. Thus, nova-compute started
causing it to behave maliciously.

\begin{figure}
\centering
\includegraphics[scale=0.4]{figures/LabAbstract-Case1}
\caption{Case One - The nova-compute service in the OpenStack-4 host is compromised.}
\label{figure:LabAbstract-Case1} 
\end{figure}
\begin{table}
\begin{tabular}{|p{.2\textwidth} | p{.8\textwidth}|}
\hline
\multicolumn{2}{|c|}{\textbf{Incident description}} \\ \hline\hline
Incident type & Malicious code and Unauthorized access\\ \hline
Current status & Ongoing attack, the malicious code is not patched nor contained yet\\ \hline
Compromised component(s) & One compute worker host\\ \hline
Physical Location & OpenStack-4 \\ \hline
Affected Layers & Cloud platform layer, the OpenStack nova-compute service \\ \hline
General Information & Malicious code is injected into the nova-compute service of the OpenStack-4 host\\ \hline
Resources at risk & Running instances on OpenStack-4, Stakeholders and resources interacting with running instances on OpenStack-4 or the infected nova-compute service \\ \hline
\end{tabular}
\caption{Case One - A compromised compute worker scenario specifications}
\label{table:case one}
\end{table}

Recommended actions by NIST and their corresponding realization in an
OpenStack deployment are explained next. They will fulfill
requirements, implied by the containment, eradication, and recovery
phase. As explained before, the described scenario consists of two
incidents, {\em unauthorized access} and {\em malicious code}. Thus,
we will in the following briefly discuss recommended responses for both types of
incident; an extended discussion can be found in \cite{aryanthesis}.

The following discussion is given in three parts. In each part, actions proposed by the NIST guideline are adapted to the cloud model.  First, containment actions will be adapted. Then, adapted form of eradication and recovery actions are explained. A major effort has been put into adapting containment actions:

\begin{itemize}
\item \textbf{"Identifying and Isolating Other Infected Hosts"}\\
\label{containment:malicious code:isolating host}
Study the profile of the infected host and compare it to other worker nodes' profiles, in order to identify compromised hosts. Comparing profiles of components is simple, using provided monitoring facilities in our experimental environment.
\item \textbf{"Blocking Particular Hosts"}\\
\label{containment:malicious code:blocking host}
The strategy should be analyzed in depth before its application. In a cloud environment when the consumer's instance is running in an infected worker node, it is not reasonable to disconnect the node without prior notice/negotiation to affected consumers (This constraint can be relaxed by providing the proper Service-level Agreement (SLA) ). 
In addition, blocking the compromised host can be done with different levels of restrictions. Initially the communication with the outside of the organization should be blocked\footnote{By the term \textit{organization}, we mean all entities who are responsible for managing the cloud infrastructure, which can be referred to as the cloud provider.}, assuming that the attacker is located outside of the organization infrastructure. Also, any further attack to the outside of the organization using compromised hosts will be mitigated. 

In the second step, communication of the compromised host with other components in the infrastructure is also restricted and the host is marked as compromised/infected/suspicious. Thus, other nodes will avoid non-critical communication with the compromised node. It will help the infrastructure to communicate with the compromised node for containment, eradication, and recovery procedures; and at the same time the risk of spreading the infection is reduced. 

The last step can be blocking the host completely. In this approach staff should access the host directly for analyzing the attack as well as assessing possible mitigation and handling strategies.

%Moreover, 
However, blocking infected hosts will not contain the incident. Each host has several consumers' instances (VM instances) and volumes running on and attached to it. Blocking hosts will only avoid spreading the incident to other hosts but instances are still in danger. An approach in a cloud environment is to disconnect instances and volumes from the underlying compromised layer. Signaling the cloud software running on the compromised host to release/terminate/shutdown/migrate instances and detach volumes are our proposed approaches. 
%A drawing of 
This approach is illustrated in Figure \ref{figure:ComputeContainment}.
We should use a quarantine compute worker node as the container for migrated instances. After ensuring the integrity and healthiness of instances, they can be moved to a regular worker node. This quarantine compute worker will be explained more in the following section.
These approaches can be implemented at the cloud infrastructure layer for simplicity (Blocking by means of nodes firewall, routers, etc.)

\begin{figure}
\centering
\includegraphics[scale=0.35]{figures/ComputeContainment}
\caption{Blocking compromised compute communication. Red lightening represent disconnected communications.}
\label{figure:ComputeContainment} 
\end{figure}

\item \textbf{"Soliciting User Participation"}\\ The interaction can
  be implemented using different methods. Distributing security
  bulletins maintained by cloud or service providers is an example of
  notifying other stakeholders about an incident. Incident or
  vulnerability reporting mechanisms are also useful when an outsider
  detects an incident or identifies a vulnerability. These two methods
  can be developed and deployed independently of the cloud
  platform. Security bulletins are provided by the security team who
  handles security related tasks. Also, reporting mechanisms are
  delivered by means of ticketing and reporting tools.  Direct and
  real-time communication among stakeholders is a complement to the above
  mentioned methods.
\item \textbf{"Disabling Services"}\\
\label{containment:malicious code:disabling services}
In order to disable a particular service, we should first check the
service dependencies diagram. An example of such a diagram is
depicted in Figure \ref{figure:ServiceDependencies}. Disabling a
service can take place in two ways.

\begin{figure}
\centering
\includegraphics[scale=0.4]{figures/ServiceDependencies}
\caption{OpenStack Nova service dependencies.}
\label{figure:ServiceDependencies} 
\end{figure}

It is possible to stop the service at the compromised host (Figure
\ref{figure:ComputeContainment2}). In our scenario we can stop the
nova-compute service to disable the compute service. It will instantly
disconnect the cloud platform from running VM instances. \textit{In
  the OpenStack platform stopping the nova-compute service will not
  terminate running instances on that host.} Thus, although the
compute service is not working anymore, already running instances will
continue to work even after  nova-compute is terminated. Additionally, it
is not possible to terminate an instance after stopping its
corresponding compute service, because the administration gateway
(i.e. nova-compute) is not listening to published messages. In order
to maintain control over running instances we should migrate instances
from the compromised node to a quarantine node before we terminate the
compute service.

\begin{figure}
\centering
\includegraphics[scale=0.35]{figures/ComputeContainment2}
\caption{Stopping the compute service at the compromised host.}
\label{figure:ComputeContainment2} 
\end{figure}

Another approach is discarding messages published by the compromised
component or those destined to it (Figure
\ref{figure:ComputeContainment3}). This is a centralized method and
the cloud controller or the messaging server should filter out
messages with the source/destination of the infected host\footnote{In
  a publisher/subscriber paradigm the destination may be eliminated or
  masked by other parameters. So, we may filter messages that contain
  any evidence of being related to the infected host.}.
\begin{figure}
\centering
\includegraphics[scale=0.35]{figures/ComputeContainment3}
\caption{Discarding messages to/from the compromised node.}
\label{figure:ComputeContainment3} 
\end{figure}
\end{itemize}
\begin{table}
\centering
\begin{tabular}{ | p{.2\textwidth} | p{.8\textwidth} | }
\hline
\textbf{NIST recommended action} & \textbf{Brief Description} \\ \hline \hline
"Identifying and Isolating Other Infected Hosts" & Extract incident symptoms to detect other infected hosts. \\ \hline
"Blocking Particular Hosts" & After identifying the compromised component and its corresponding host (i.e. the compromised worker/compute host), that host should be blocked. \\ \hline
"Soliciting User Participation" & Interaction among cloud stakeholders (e.g. cloud providers, cloud consumers, third parties, end users, etc.) is a mandatory step toward fulfilling incident containment requirements.\\ \hline 
"Disabling Services" & Disabling the infected service (nova-compute in our scenario) may reduce impacts of the compromised host. Disabling a service can disrupt other services and cause deviation from promised SLA by the provider. \\ \hline 
\end{tabular}
\caption{Containment Strategies}
\label{table:Containment Strategies}
\end{table}

%We explained four actions for containing a malicious code incident. 
We continue by explaining four other actions which are recommended
responses to an unauthorized access incident:
\begin{itemize}
\item \textbf{"Isolate the affected systems"}\\
The same procedures as those which have been explained for "Identifying and Isolating Other Infected Hosts" (Section \ref{containment:malicious code:isolating host}) and "Blocking Particular Hosts" (Section \ref{containment:malicious code:blocking host}) can be applied here.
\item \textbf{"Disable the affected service"}\\
The same procedure as the one which has been explained for "Disabling Services" (Section \ref{containment:malicious code:disabling services}) can be applied here.
\item \textbf{"Eliminate the attacker's route into the environment"}\\
Access methods which have been used by the attacker to access cloud components should be blocked. Implementing filtering mechanisms in the messaging server is a crucial requirement which is highlighted in different strategies. The cloud provider should be capable of blocking messages which are related to the attack and blocks the attacker's route into the cloud environment.
It should be noted that the mechanisms which we have used to meet requirements imposed by "Blocking Particular Hosts", "Identifying and Isolating Other Infected Hosts", "Disabling Services" (Section \ref{containment:malicious code:disabling services}) are appropriate actions for eliminating attackers' routes.
\item \textbf{"Disable user accounts that may have been used in the
  attack"}\\ 
A compromised user account may reside in multiple layers,
  such as the system, cloud platform, or VM instance layer\footnote{It
    should be noted, although we may use directory and federation
    services to unify users among services and layers, this may not be
    a feasible %nor plausible 
approach in a cloud environment. However,
    federation is applicable at each layer (e.g. system, cloud
    platform, VM instances).}. Based on the membership layer, the
  disabling and containment procedure will differ. Additionally, in
  each layer a variety of user types exist. As an example, in the
  cloud platform layer, the cloud provider's staff and cloud consumers
  have a different set of user types.
\end{itemize}


As it was explained, three phases are adapted. Containment phase was discussed, and eradication phase is the next one to be studied:
% rephrase ?
\begin{itemize}
\item \textbf{"Disinfect, quarantine, delete, and replace infected
  files"}\\ 
These strategies are applicable in two layers depending on
  the container of the injected malicious code. The malicious code can
  be injected into either the cloud platform services
  (i.e. nova-compute) or the OS modules/services.  If the injected
  malicious code is in OS modules/services, utilizing existing
  techniques is effective. By existing techniques, we refer to anti
  virus software and traditional malware handling mechanisms. In this
  case nothing new has happened, although side effects of the incident
  may vary a lot.

However, if the malicious code is injected into a cloud platform service (in our case nova-compute), existing anti virus products are not useful, as they are not aware of the new context. Cleaning a cloud platform service can be very hard, so other approaches are more plausible. In general, we can propose several approaches for eradicating a malicious code incident in a cloud platform:
\begin{itemize}
\item Updating the code to the latest stable version and apply appropriate patches to fix the vulnerability.
\item Purging the infected service on the compromised node 
\item Replacing the infected service with another one that uses a different set of application layer resources (e.g. configuration files, repositories, etc.) 
\end{itemize}

It should be noted that in a highly distributed system such as a cloud environment, doing complicated tasks such as fixing a single infected node in real time fashion does not support the cost effectiveness policy. Thus, terminating the infected service or even the compromised node and postponing the eradication phase can be an appropriate strategy.
\item \textbf{"Mitigate the exploited vulnerabilities for other hosts within the organization"}\\
In order to complete the task, we should also update the cloud platform software on other nodes and patch identified vulnerabilities.
\end{itemize}

The last phase is about recovery of the system which was under attack:
%  rephrase? 
\begin{itemize}
\item \textbf{"Confirm that the affected systems are functioning normally"}\\
Profiling the system is useful in the recovery phase as well as in the detection and analysis phase. After containment and eradication of the compromised component, the component profile should be the same as a healthy component or be the same as its own profile before being infected. Using the provided tools in our deployment (i.e. Cacti) we can specify the exact period and components which we want to compare.
\item \textbf{"If necessary, implement additional monitoring to look for future related activity"}\\
After identifying attack patterns and the compromised node profile, we should add proper monitoring alarms to cover those patterns and profiles. As an example, if the compromised compute worker starts to request 
%for 
a large number of IP addresses after its infection, this pattern
should be saved and monitored on other compute workers. So, if we
experience a compute worker with the same profile and behavior, that
worker node will flagged as possibly 
%become suspicious for being 
infected.  In our
monitoring tools, the administrator can define a threshold for different
parameters; if the current profile of the system violates the
threshold, graphs will be drawn with %other 
a different color to notify the
user. We can also add other monitoring tools to generate the ticket in
case of a matching profile.
%%% ???? Don't understand this:
%, that is not required yet.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Case Two: A bogus component}
A bogus service is a threat to %for the cloud environment security. As
OpenStack is an open source software, an attacker can access the
source code or its binaries and start a cloud component that delivers
a specific service. When the attacker is managing a service, he/she
can manipulate the service in a way that threatens the integrity and
confidentiality of the environment. This section will discuss such an
incident, where 
%that 
a bogus nova-compute service 
%cloud platform component 
is added to the
cloud environment. 
%We will focus on a nova-compute service as the bogus
%component in the cloud environment. 
The incident description for this
case is given in Table \ref{table:case two}

A bogus nova-compute service (or, in general, any cloud platform
component) can run on a physical machine or a virtual instance.  
It is unlikely that an 
attacker will be capable of adding a physical node to the cloud infrastructure; 
%unlikely; 
however, for the sake of completeness we study both the case
that the bogus service is running on a new physical machine and the
one where it is running on a virtual instance. Both cases are depicted
in Figures \ref{figure:LabAbstract-Case2} and
\ref{figure:LabAbstract-Case2-Instance}.

\begin{table}
\centering 
\begin{tabular}{ | p{.2\textwidth} | p{.8\textwidth} | }
\hline
\multicolumn{2}{|c|}{\textbf{Incident description}} \\ \hline\hline
Incident type & Inappropriate Usage \\ \hline
Current status & Ongoing attack, the bogus compute worker is still up and serving a part of requests\\ \hline
%Compromised component(s) & One compute worker host\\ \hline
Physical Location & OpenStack-5 \\ \hline
Affected Layers & Cloud platform layer, the OpenStack nova-compute service, consumers' instances \\ \hline
General Information & A bogus compute worker node is added to the platform, it is a threat to the provider's and consumers' data confidentiality and integrity. Also a threat for the system availability. \\ \hline
Resources at risk & Running instances on OpenStack-5, Stakeholders and resources interacting with running instance on OpenStack-5 \\ \hline
\end{tabular}
\caption{Case Two - A bogus component scenario specifications}
\label{table:case two}
\end{table}
\begin{figure}
\centering
\includegraphics[scale=0.3]{figures/LabAbstract-Case2}
\caption{Case Two - A physical bogus compute worker node is added to the infrastructure.}
\label{figure:LabAbstract-Case2} 
\end{figure}
\begin{figure}
\centering 
\includegraphics[scale=0.4]{figures/LabAbstract-Case2-Instance}
\caption{Case Two - A virtual bogus compute worker is added as a consumer's instance.}
\label{figure:LabAbstract-Case2-Instance} 
\end{figure}

When the bogus service is running on top of an instance, the network
connectivity may be more limited than compared to the other case
(i.e., the bogus service is running on a physical node). Initially,
any given instance is only connected to the second interface
(\textsl{eth1}). This connectivity is provided by means of the bridge
connection (\textsl{br100}) that connects virtual interfaces
(\textsl{vnetX}) to the rest of the environment. Thus, a running
instance has no connectivity to the switch \textsl{SW2} by default.  However,
connectivity to the outside world can be requested by any consumer
(e.g., an attacker) through a legitimate procedure. Thus, in Figure
\ref{figure:LabAbstract-Case2-Instance}, we also connect the instance
to  \textsl{SW2}.

We simulated the virtual bogus compute worker by deploying the
nova-compute service on a running instance. There were multiple
obstacles for simulating this scenario, including: the running
instance, which turns out to be also a bogus worker, must have %the
hosting capabilities; the bogus worker must respond to cloud
controller requests to be recognized as a working node.

Detecting a bogus worker node or instance is a complex task if the
infrastructure has not previously employed a proper set of
mechanisms. However, a few parameters can be monitored as an
indication of a bogus worker.  Generally, a bogus worker is not
working as well as a real one, because its main goal is not providing
a regular service. A bogus worker aims to steal consumers' data,
intrude on the cloud infrastructure, disrupt the cloud environment
Quality of Service (QoS), and so forth. Without any prior preparation,
a suspicious worker can be identified by monitoring the service
availability and QoS parameters on each worker. Moreover, a suspicious
virtual worker can also be recognized because of its high traffic
towards the cloud infrastructure messaging servers.

Containing a bogus worker consists of both proactive and reactive
techniques. When a bogus worker is detected, the containment procedure
is fairly simple (i.e., applying reactive techniques). However,
deploying a set of proactive techniques is more challenging. These
techniques can be implemented as a group of security mechanisms and
policies, such as \textit{node authentication}, \textit{manual
  confirmation}, \textit{trust levels and timeouts}, and \textit{no new
  worker policy}. They will be discussed further in Section
\ref{approaches:policies}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Approaches for Containment and Recovery}\label{section:approaches}
%% \textit{\textbf{TODO:} Intro...}
%% \begin{enumerate}[A.]
%% \item Restricting infected components
%% \item Replicating services
%% \item Disinfecting infected components
%% \item Migrating instances
%% \item Node authentication
%% \item Policies
%% \end{enumerate}
This section introduces our proposed approaches for containment of
intruders, eradication of malicious processes and recovery from
attack. The proposed strategies can be grouped based on two criteria: The
responsible stakeholder for developing and deploying the strategy, and
the target layer for that strategy. Based on the first criterion we
may have either the cloud provider or the cloud consumer as the responsible
stakeholder; based on the second criterion, the target layer can
be either the infrastructure/hardware layer or the service/application layer.
We have devised a set of approaches which will be explained in detail
in the following.
%Following sections will explain each approach in detail:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Restricting infected components
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Restriction, disinfection, and replication of infected cloud platform components}
A general technique for containing an incident is restricting the
infected component. The restriction can be applied in different
layers, with a variety of approaches, such as: filtering in the AMQP
server, filtering in other components, disabling the infected service
or disabling the communicator service. Additional measures can also be
employed to support the restriction, like: removing infected instances
from the project VLAN, disabling live migration, or quarantining
infected instances.  We explain each of these approaches in the
following sections.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Filtering in the messaging server (cloud controller)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Filtering in the messaging server (cloud controller)} \label{approach:msg_srv_filter}
We will propose several filtering mechanisms in the messaging server
in order to contain and eradicate an incident in a cloud
environment. The OpenStack platform has been used to build our
experimental cloud environment. This approach is a responsibility of
the cloud provider and the target layer in the cloud platform
application layer.
\paragraph{Advantages}
%The filtering in the messaging server has its own advantages and disadvantages, which will be discussed next.
\begin{itemize}
\item The filtering task at the messaging server level can be done
  without implementation of new functionality. We can use existing
  management interfaces of the RabbitMQ (either command line or web
  interface) to filter the compromised component.
%% However, in a large scale deployment of the platform, the
%% situation may vary. When automation and real-time responses
%% are crucial, we have to avoid mechanisms which require
%% human intervention. Even in this case we should only
%% implement a set of functionalities that uses management
%% interfaces for filtering. Thus, instead of an operator who
%% terminates a connection manually, the cloud controller will
%% do that when it is required.
\item The filtering task can be done in a centralized fashion by means of the management plug-in, although we may have multiple instances of the messaging server.
\item Implementing this approach is completely transparent for other stakeholders, such as cloud consumers.
\item We can scale out\footnote{Scaling out or horizontal scaling is
  referred to the application deployment on multiple servers
  \cite{4228359}.} the messaging capability by running multiple
  instances of the RabbitMQ on different nodes. Scaling out the
  messaging server will also scale out the filtering
  mechanism\footnote{But it may require a correlation entity to handle
    the filtering tasks among all messaging servers.}.
\item This approach is at the application layer, and it is independent of network architecture and employed hardware.
\item The implementation at the messaging server level helps in having a fine-grained filtering, based on the message content.
\end{itemize}

\paragraph{Disadvantages}
\begin{itemize}
\item A centralized approach %has its own disadvantages as well, such as being 
implies the risk of a single point of failure or becoming the system bottleneck.
\item Implementing the filtering mechanism at the messaging server and/or the cloud controller adds an extra complexity to these components.
\item When messages are filtered at the application layer in the
  RabbitMQ server, the network bandwidth is already wasted for the
  message that has an infected source, destination, or even
  context. Thus, this approach is less efficient %comparing to the
  than one that may filter the message sooner (e.g. at its source
  host, or in the source cluster).
\item Most of the time application layer approaches are not as fast as
  those in the hardware layer. %one.  
In a large scale and distributed
  environment the operation speed plays a vital role in the system
  availability and QoS.  It is possible to use the zFilter technique \cite{Jokela:2009:LLS:1592568.1592592}
  as a more efficient implementation of the message delivery
  technique. It can be implemented on either software or hardware. The
  zFilter is based on the bloom-filter \cite{Broder02networkapplications}
data structure. Each message
  contains its state; thus this technique is stateless
  \cite{Jokela:2009:LLS:1592568.1592592}. It also utilizes source
  routing. zFilter implementations are available for the BSD family
  operating systems and the NetFPGA boards at the following address,
  \textsl{http://www.psirp.org}.
\item Filtering a message without notifying upper layers may lead to
  triggered timeouts and resend requests from waiting entities. It can
  also cause more wasted bandwidth.
\end{itemize}

\paragraph{Realization}

A variety of filtering mechanisms can be utilized in the messaging server; each of these mechanisms focuses on a specific component/concept in the RabbitMQ messaging server. We can enforce the filtering in the messaging server \textit{connection}, \textit{exchange}, and \textit{queue} %that 
as will be discussed next.
\begin{itemize}
\item \textbf{Connection:} A connection is created to connect a client
  to an AMQP broker \cite{rabbitmq:admin-guide}. A connection is a
  long-lasting communication capability and may contain multiple
  channels \cite{amqp0-8}. By closing the connection, all of its
  channels will be closed as well. A snapshot of connections in our
  OpenStack deployment is available in Figure
  \ref{figure:RabbitMQConnections}.
\begin{figure}
\centering
\includegraphics[scale=0.4]{figures/RabbitMQConnections}
\caption{RabbitMQ Connections} 
\label{figure:RabbitMQConnections}
\end{figure} 
% First approach to block the compromised component is closing
% its client connection. Closing the connection will stop all
% channels in that connection.
\item \textbf{Exchange:} An exchange is a message routing agent which
  can be durable, temporary, or auto-deleted. Messages are routed to
  qualified queues by the exchange. A Binding is a link between an
  exchange and a queue. An exchange type can be one of \textit{direct,
    topic, headers, } or \textit{fanout} \cite{rabbitmq:introduction}.
  An exchange can be manipulated in different ways in order to provide
  a filter mechanisms for our cloud environment:
\begin{itemize}
\item \textbf{Unbinding a queue from the exchange:} The compromised
  component queue won't receive messages from the unbound exchange. As
  an example, we assume that the compute service of the OpenStack-4
  host is compromised. Now, we want to block nova traffic to and from
  the compromised compute service; so, we unbind the \textsc{nova}
  topic exchange from the queue
  \textsc{compute.openstack-4}. 
%Provided
The RabbitMQ management interface is used to unbind the exchange, as
shown in Figure \ref{figure:RabbitMQUnbindingExchange}.
\begin{figure}
\centering
\includegraphics[scale=0.4]{figures/RabbitMQUnbindingExchange}
\caption{Unbinding a queue from an exchange using the Queues Management page of RabbitMQ}
\label{figure:RabbitMQUnbindingExchange} 
\end{figure}
\item \textbf{Publishing a warning message:}
Publishing an alert message to that exchange, so all clients using that exchange will be informed about the compromised component. Thus, by specifying the compromised component, other clients can avoid communicating with it. The main obstacle in this technique is the requirement for implementing new functionalities in clients.
\item \textbf{Deleting the exchange:}
Deleting an exchange will stop routing of messages related to it. It may have multiple side effects, such as memory overflow and queue exhaustion. 
\end{itemize}
\item \textbf{Queue:}
The queue is called 
%as 
a "weak FIFO" buffer;
% that 
each message in it can be delivered only to a single client unless re-queuing the message \cite{rabbitmq:introduction}.
\begin{itemize}
\item \textbf{Unbinding} a queue from an exchange avoids further
  routing of messages from that exchange to the unbound queue. We can
  unbind the queue which is connected to the compromised component and
  stop receiving messages by the infected client.
\item \textbf{Deleting}
a queue not only removes the queue itself, but also remove all messages in the queue and cancel all consumers on that queue. 
\item \textbf{Purging} a queue removes all messages in the queue that
  do not need acknowledgment. Although it may be useful in some cases,
  it may not be as effective as required
%in occurrence of 
during an incident.
\end{itemize}
Figure \ref{figure:RabbitMQInternal}
%\footnote{Multiple details have been avoided in this figure to make
%it more readable, such as Virtual Host.} 
depicts a simplified overview of messaging server internal entities and the application points of our approaches.
\begin{figure}
\centering
\includegraphics[scale=0.5]{figures/RabbitMQInternal}
\caption{Overview of RabbitMQ messaging server and applicable containment approaches. }
\label{figure:RabbitMQInternal} 
\end{figure}
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Filtering in each component}\label{approach:component_filter}
Applicable filtering mechanisms in the messaging server have been studied in the previous section. This section discusses mechanisms that are appropriate for other components. These components are not essentially aware of messaging technique details and specifications.
\paragraph{Advantages}
%Advantages of filtering messages in each components include:
\begin{itemize}
\item The implementation of the filtering mechanism in each component avoids added complexity to the messaging server and cloud controller.
\item This approach is a distributed solution without a single point
  of failure, in contrast to the previous one with a centralized
  filtering mechanism.
\item Assuming the locality principle in the cloud, wasted bandwidth is
  limited to a cluster/rack which hosts the infected
  components. Network connections have much higher speed in a rack or
  cluster.
\item This approach does not require a correlation/coordination entity
  for filtering messages. Each component behaves independently and
  autonomously upon receiving an alarm message
%, that 
which announces a
  compromised node.  Traditionally, most security mechanisms have been employed
  at the organization/system boundaries. 
%However, as the realization
 % of boundaries is becoming weaker in a cloud environment, this
  %approach is a reasonable one to fulfill the new requirements.
However, as there is no boundary in the cloud, performing
  security enforcement at each component is a more reliable
  approach. 
\end{itemize}
\paragraph{Disadvantages}
%And its disadvantages are:
\begin{itemize}
\item When the filtering must be performed in each component, all
  interacting components must be modified to support the filtering
  mechanism. However, this issue can be relaxed by using a unified
  version of messaging client (e.g., pika python client) and modifying
  the client in case of new requirements.
\item The message which should be discarded traverses all the way down to the destination, and wastes the link bandwidth on its route.
\item Dropping a message without notifying upper layers, may lead to triggered timeouts  and resend requests from waiting entities. It can also cause more wasted bandwidth.
\end{itemize}

\paragraph{Realization}

This approach can be implemented at two different levels: blocking at either the messaging client level (e.g. AMQP messaging client) or the OpenStack component/service level.

First, the responsible client can be modified to drop messages with
specific properties (e.g. infected source/destination). As an example,
the responsible client for AMQP messaging in  OpenStack is
amqplib/pika; we must implement the mechanism in this AMQP client (or
its wrapper in OpenStack) to filter malicious AMQP messages. Using
this method, more interaction between OpenStack and clients may be
required to avoid resend requests. Because of using the same AMQP
client in all components, the implementation is easier and 
%its
the modification process 
%needs 
requires less effort. The second method is to
develop 
%the 
filtering in each of the OpenStack components, such as
nova-compute, nova-network, nova-scheduler, etc. This method adds more
complexity to those components and it may not be part of their
responsibilities.

We propose a combination of these methods. Implementing the filtering
mechanism in the carrot/amqplib wrapper of 
%the 
OpenStack has
advantages of both methods, and avoids unnecessary complexity. The
OpenStack wrapper for managing AMQP messaging is implemented in
\textsl{src/nova/rpc.py}. In order to identify the malicious message,
we use the message address which is part of its context. Then, the
actual dropping happens in the \textsl{AdapterConsumer}
method. Assuming that the source address is set in the context
variable, filtering is straightforward. By checking the message
address and avoiding the method call, most of the task is done. The
only remaining part is to inform the sender about the problem, this
can be implemented by means of the existing message reply
functionality.
%In addition to this modification another feature should be added to
%handle the list of compromised components.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Disabling services}\label{approach:disabling_service}
Disabling services is a strategy for containing the incident. The
disabled service can be either the infected service itself or the communicator
%one. 
service. The 
%communicator service 
latter handles task distribution and
delegation.
This method can be used only by the cloud provider, and is at the application layer.
\paragraph{Disabling an infected service}
An incident can be contained by disabling the infected service. It has several advantages, including:
\begin{itemize}
\item After the nova-compute service is stopped, running instances will continue to work. Thus, as a result consumers' instances will not be terminated nor disrupted.
\item All communications to and from the compromised node will be stopped. So, the wasted bandwidth will be significantly reduced. 
%massively.
\item Shutting down a service gracefully avoids an extra set of
  failures. When the service is stopped by the Nova interfaces, all other
  components will be notified and the compromised node will be removed
  from the list of available compute workers.
\end{itemize}
Like any other solution, it has multiple drawbacks as well, including:
\begin{itemize}
\item Keeping instances in 
%the 
a running state can threaten other cloud consumers. The attacker may gain %an
access to running instances on the compromised node.
\item The live migration feature will not work anymore. Thus, the threatened consumers cannot migrate running instances to a safe or quarantine compute worker node.
\item Neither the cloud provider nor consumers can manage running instances through the OpenStack platform.
\end{itemize}
This approach requires no further implementation, although we may like to add a mechanism to turn services on and off remotely.
\paragraph{Disabling a communicator service}
An incident can be contained by disabling or modifying its
corresponding communicator service. An example of a communicator
service in an OpenStack deployment nova-scheduler service. The
nova-scheduler decides
%that 
which worker should handle 
%the 
a newly arrived request, such as running
an instance.  By adding new features to the scheduler service, the
platform can avoid forwarding requests to the compromised node.
Advantages of this approach are:
\begin{itemize}
\item No more requests will be forwarded to the compromised node.
\item Consumers' instances remain in the running status on the
  compromised node. So, consumers will have enough time to migrate
  their instances to a quarantine worker node or dispose of their
  critical data. 
%%%%%%% ????
% Even estimate impacts of the incident.
\item This approach can be used to identify the attackers, hidden system vulnerabilities, and the set of employed exploits. In other words, it can be used for forensic purposes.
\end{itemize}
Disadvantages of disabling communication include:
\begin{itemize}
\item New features 
%should 
must be implemented. These new features are more focused on the
decision algorithm of the scheduler service.
\item This approach will not secure the rest of our cloud environment, but it avoids forwarding new requests to the compromised node. However, this drawback can be seen as an opportunity. We can apply this approach and also move the compromised node to a \textbf{HoneyCloud}. In the HoneyCloud we don't restrict the compromised node, instead analyze the attack and attacker's behavior. But even by moving the compromised node to a HoneyCloud, hosted instances on that node are still in danger. 

It is possible that consumers' instances are all interconnected. Thus,
those running instances on the compromised node in the HoneyCloud
could threaten the rest of the consumers' instances. The rest of the 
instances may even be hosted on a secure worker node. The next
proposed approach is a solution for this issue.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Replicating services}\label{approach:replicate_service}
An approach to overcome the implications of an incident is replicating
services. A service in this 
%section 
context is a service which is delivered
and maintained by the cloud provider. It can be a cloud platform
service (e.g nova-compute) or any other services that concern other
stakeholders. The replication can be done passively or actively, and
that is due to new characteristics of the cloud model. The replication
of a cloud service can be done either at the physical or virtual
machine layer.

Replicating a service on physical machines is already done in platforms such as 
OpenStack. The provider can replicate cloud services either passively or actively when facing an issue in the environment. 

Replication of a service on virtual machines has multiple benefits, including: 
\begin{itemize}
\item Virtual machines can be migrated while running (i.e. live migration), this is a practical mechanism for stateful services that use memory.
\item Replication at the instance layer is helpful for forensic purposes. It is also possible to move the compromised service in conjunction with the underlying instance to a HoneyCloud. This is done instead of moving the physical node, ceasing all services on it, and changing the network configuration in order to restrict the compromised node communication.
\item By using virtual machines in a cloud environment we can also benefit from the cloud model elasticity and on demand access to computing resources.
\end{itemize}
This approach is also the main idea behind the \textit{Virtualization Intrusion Tolerance Based on Cloud Computing} (CC-VIT) \cite{5678134}. By applying the CC-VIT to our environment, the preferred hybrid fault model will be Redundant Execution on Multiple Hosts (REMH), and the group communication is handle using the AMQP messaging.
We can use physical-to-virtual converters to have the advantages of both approaches. These tools convert a physical machine to a virtual machine image/instance that can be run on top of a hypervisor.
Moreover, each of these replicas can be either active or passive. This will have a great impact on the system availability. 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Disinfecting infected components}\label{approach:disinfecting_component}
Disinfecting an infected component is a crucial task in handling an incident and securing the system. It can be accomplished with multiple methods having a variety of specifications.
None of the following approaches will be used for cleaning the infected binary files, instead less complex techniques are employed that can be applied in a highly distributed environment. Cleaning a binary file can be offered by a third party security service provider, 
%that has focused on large scale antivirus software.
but that will not be discussed further here. 
\begin{enumerate}
\item \textbf{Updating the code}\\ The service code can be updated to
  the latest, patched version. This process should be done in a smooth
  way so that all components will be either updated or remain
  compatible with each other after a partial component update.  Several
  tools has been developed
%with 
for this purpose; one of the best examples is the Puppet project \cite{puppet}.
\item \textbf{Purging the infected service}\\ Assuming that the
  attacker has stopped at the cloud platform layer, we can 
%assure 
ensure containment of the incident by removing the
  service completely.
\item \textbf{Replacing the service}\\ Another method which is not as
  strong as the others is achieved by replacing the infected service
  with another one that uses a different set of application layer
  resources, such as configuration files, binaries, etc. Thus, we can
  be sure that the infected resources have no effect on the new
  service.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Isolation, disinfection, and migration of instances}
In the following part, those techniques are discussed which are handling virtual machine instances. Three major approaches can be chosen for handling an attacked instance: isolating, disinfecting, and migrating a given one. Each of them will be explained next.


\subsubsection{Removing instances from the project VLAN}\label{approach:removing_vlan}
This approach does not contain the compromised node, instead it
focuses on containing instances hosted by the compromised worker
node. This is important because those instances may have been
compromised as well. The first step toward securing the consumer's
service is to disconnect potentially infected instances.  The main
usecase of this approach is when the attacker disrupts other solutions
(e.g.,  disabling nova-compute management functionalities through escalated
privileges at the OS layer), or when instances and the consumer's
service security is very important (e.g., eGovernment services).  It
has several advantages specifically for cloud consumers, including:
\begin{itemize}
\item It can disconnect potentially infected instances from the rest of the consumer's instance.
\item It does not require implementation of new features.
\item The attacker cannot disrupt this method.
\end{itemize}
The disadvantages are as follows:
\begin{itemize}
\item This method only works in a specific OpenStack networking mode
  (i.e., the  VLANManager networking mode).
\item The consumer completely loses control over isolated instances,
  this may lead to data loss or disclosure, service unavailability,
  etc.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Disabling live migration}\label{approach:disble_live_migration}
Live migration can cause widespread infection, or can be a mechanism
for further intrusion to a cloud environment. It may take place
intentionally or unintentionally (e.g., an affected consumer may
migrate instances to resolve the attack side effects, or the attacker
%that has the 
with consumer privileges migrates instances to use a
hypervisor vulnerability and gain control over more nodes). Disabling
this feature helps the cloud provider to contain the incident more
easily, and keep the rest of the environment safer.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Quarantining instances}\label{approach:quarantine}
When we migrate instances from a compromised node, we cannot accept the risk of spreading infection along instance migration. Thus, we should move them to a quarantine worker node first. The quarantine worker node has specific functionalities and tasks, including:
\begin{itemize}
\item This worker node limits instances' connectivity with the rest of
  cloud environment. As an example, only cloud management
  requests/responses are delivered by the quarantine host.
\item It has a set of mechanisms to check instances' integrity and healthiness. These mechanisms can be provided by the underlying hypervisor, cloud platform, or third parties' services.
\end{itemize}
In order to deploy a quarantine node, 
%we should study and employ a
a set of mechanismswe should be studied and employed. 
%In most cases, we will introduce the appropriate
Tools that implement such mechanisms will be presented below.
\begin{enumerate}
\item \textbf{Virtual Machine Introspection}\\ This mechanism
  simplifies introspecting the memory space of a virtual machine from
  another virtual machine. The task is fairly complex because of the
  semantic gap between the memory space of those two virtual machines.
%%%%%%%%%%% Ref?  
XenAccess \cite{xenaccess} is an example of an introspection library. Using XenAccess
  the privileged domain can monitor another Xen domain.
\item \textbf{Domain Monitoring}\\
One of the basic methods to identify a compromised instance is by means of profiling and monitoring the instance behavior. Domain monitoring techniques provide an abstract set of data, compared to the detailed, low level output of a VM introspection tool.
For a virtual machine running over a Linux box we can use the libvirt \cite{libvirt} library to access the suspicious instance and study its behavior.
\item \textbf{Intrusion Detection}\\ Having an intrusion detection
  system in the hypervisor or cloud platform layer not only provide
  better visibility for security mechanisms but is also more resistant
  against a targeted attack from an unauthorized access to an
  instance. Livewire \cite{garfinkel:vmi} is a prototype
  implementation of an intrusion detection system in a hypervisor.
  Another way to benefit from an intrusion detection system is 
%the same
  Amazon's approach, which offers you a standalone Amazon Machine Image (AMI) that contains Snort and Sourcefire Vulnerability Research Team rules. The consumer can then forward its instances' traffic to the virtual machine
  with intrusion detection capabilities. The same approach can be utilized
  in our deployment. The main issue is the approach's performance and
  utilization.
\item \textbf{Utilizing trusted computing concepts}\\
Trusted computing is a technology for ensuring the confidentiality and integrity of a computation. 
%Moreover 
It is also useful for remote attestation. Thus, we can use the
technology not only for securing our deployment but also to build a
better quarantine and infection analysis mechanism.  
%Several
Approaches that have used this concept include vTPM: Virtualizing the
Trusted Platform Module \cite{vTPM}, TCCP: Trusted Cloud Computing
Platform \cite{Santos09towardstrusted}, and TVDc: IBM Trusted Virtual
Datacenter \cite{TVDc}.
\end{enumerate}
It should be noted that although cloud providers or third-party
service providers can offer an IDS agent service inside each instance,
they cannot force the consumer into accepting it. It is a reasonable
argument due to the consumer's organization internal security policies and
resource overhead because of the security agent. Thus, applying
security services to the underlying layer (i.e. hypervisor, cloud
platform) is a preferred solution.  Detailed specifications of such a
compute worker node is a great opportunity for future work.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% WHAT IS TODO HERE?
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Recovering an instance}\label{approach:disinfect_instance}
Recovering an infected or malfunctioning instance can 
%happen 
be performed using
different techniques. An instance can either be disinfected internally
or rebooted from a clean image. However, a tight collaboration between
provider and consumer is required for any of these techniques.

Obviously, disinfection of an instance cannot be performed solely by
the provider. Because it should not access the instance internally, but
can only provide a disinfection service (e.g. instance anti-virus) to
be used by the consumer at its own will. On the other hand, rebooting
an instance from a clean image can be done by the provider or the
consumer. Nevertheless, there are several issues in performing the
reboot action. First, one must make sure that the instance termination
will be done gracefully, so no data will be lost. Second, the VM image
must be analyzed for any flaws or security vulnerabilities. Third,
before attaching the storage to the rebooted instance, the volume must
be disinfected.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Migrating instances}\label{approach:migrate_instance}
The affected consumer can migrate a specific instance or a set of
instances to another compute worker or even another cloud
environment. The migration among different providers is currently an open
challenge, because of the weak interoperability of cloud
systems and lack of standard interfaces for cloud services.  In our
deployment, both Amazon EC2 APIs and Rackspace APIs are
supported. Thus, in theory a consumer can move between any cloud
environment provided by the Amazon EC2, RackSpace, and any open
deployment of OpenStack without any problem.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Policies}\label{approaches:policies}
In addition to all techniques that have been studied, a group of security policies should be developed and exercised. These policies can be implemented and enforced inside those techniques, as additional measures.

\subsubsection{Component authentication}\label{approach:component_auth}
The component authentication policy enforces that each worker must have a certificate signed by a trusted authority. This authority can be either an external one or the cloud controller/authentication manager itself. Having a signed certificate, the worker can communicate with other components securely. The secure communication can bring us any of the following: confidentiality, integrity, authentication, and non-repudiation.

In this case, the worker's communication confidentiality and
authenticity is important for us. For this purpose we can use two
different schemes: message encryption or a signature scheme. Each of
these schemes can be used for the whole communication or the handshake
phase only.  When any of those schemes are applied only to the
handshake phase, any disconnection or timeout in the communication is
a threat to the trust relation. As an authenticated worker is
disconnected and reconnected, we cannot only rely on the worker's ID
or host-name to presume it as the trusted one. Thus, the handshake
phase should be repeated to ensure the authenticity of the worker.
Although applying each scheme to all messages among cloud components
is tolerant against disruption and disconnection, its overhead for the
system and the demand for it should be studied case by case.  By
applying each of those schemes to all messages, we can tolerate
disconnection and disruption. However, using cryptographic techniques
for all messages introduce an overhead for the system which may not be
efficient or acceptable.

Implementing this method in our environment is simple. The RabbitMQ has features that facilitate communication encryption and client authentication. The \textsl{RabbitMQ SSL support} offers encrypted communication \cite{rabbitmq:ssl}. 
Moreover, an authentication mechanism using the client SSL certificate is offered by the \textsl{rabbitmq-auth-mechanism-ssl} plugin \cite{rabbitmq:auth}.

\subsubsection{No new worker policy}\label{approach:no_new_worker}
In addition to 
%all those 
the previously discussed technical approaches, a set of management
policies can also relax the issue. As an example, no new worker should
be added unless there is a demand for it. The demand for a new worker
can be determined when the resource utilization for each zone is above
a given threshold.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Trust levels and timeouts}\label{approach:trust_levels}\label{approach:manual_confirmation}

Introducing a set of trust levels, a new worker can be labeled as a
not trusted worker. Workers which are not trusted yet, can be used for
hosting non-critical instances, or can offer a cheaper service to
consumers.  In order to ensure the system trustworthiness in a long
run, a not-trusted worker will be disabled after a timeout. A simple
Finite State Machine (FSM) model of those transitions is depicted in Figure
\ref{figure:TrustMarkov1}.
\begin{figure}
\centering 
\includegraphics[scale=0.6]{figures/TrustMarkov1}
\caption{A simple Finite State Machine (FSM) model for trust states of a component.}
\label{figure:TrustMarkov1} 
\end{figure}
Assuming we have only two trust levels, Figure \ref{figure:TrustMarkov2} depicts transitions between them. As an example, \textsl{T0} can be achieved by human intervention; and the second level of trust \textsl{T1} is gained by cryptographic techniques or trusted computing mechanisms.
\begin{figure}
\centering 
\includegraphics[scale=0.6]{figures/TrustMarkov2}
\caption{A simple FSM model for transitions between different trust levels of a component.}
\label{figure:TrustMarkov2} 
\end{figure}

This policy can be implemented in the cloud platform scheduler
(e.g. nova-scheduler is the responsible component in the OpenStack
platform). Implementing this policy will 
%let 
allow the cloud provider to
offer more resources for non-critical use-cases. However, the offered QoS
might not be as good as before. The effectiveness of this approach is
highly dependent on a few parameters, such as the ratio of adding new
workers, trust mechanisms and their performance, and consumers'
use-cases and requested QoS.

A major challenge in this approach is about trust mechanisms. The simplest
mechanisms will be manual determination and confirmation of
authenticity and trust level. A recently added worker won't be used
for serving consumers' requests until its authenticity is confirmed by
the relevant authority (e.g. cloud provider). However, this does not
scale: The human intervention can simply become a bottleneck in the
system.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Comparison}\label{subsection:comparison}
A list of security mechanisms have been discussed. Although most of
them are orthogonal to each other, they can be compared in terms of
their common criteria (Table \ref{table:comparison}). A few criteria are extracted and explained in
the following part.
% Then  is provided that compares approaches against their common criteria.

\begin{itemize}
	\item \textbf{Responsible stakeholder}: Each mechanism must be
          delivered by a single stakeholder or a group of
          stakeholders. Identifying those responsibilities and
          assigning them to the right bodies is a crucial step toward
          building a secure environment.

	\item \textbf{Proactive/Reactive}: Both proactive and reactive mechanisms have been discussed above. Knowing mechanisms' behavior is useful in their enforcement and
comparison.
	
	\item \textbf{Service impact (Affected entities)}: Inevitably, enforcing each mechanism will introduce a set of side effects to delivered services and working entities. Identifying these side effects makes the enforcement process much more predictable.
	
	\item \textbf{Implementation/Enforcement difficulties}: Finding out implementation challenges of security mechanisms is important. These challenges are meaningful measures in comparing mechanisms with each other.
	
	\item \textbf{Dependencies}: Dependencies of approaches make
          them bounded to a specific platform and libraries. Having
          less or looser dependencies makes the solution more
          portable. Portable approaches can be developed as generic
          services that can be applied to a variety of
          platforms. Thus, we can see the importance of having common
          and standard interfaces among different platforms, such as
          Open Cloud Computing Interface (OCCI) and Cloud Data
          Management Interface (CDMI).
\end{itemize}

\begin{sidewaystable}
%\begin{table}
\begin{tabular}{| p{.15\textwidth} | p{.025\textwidth} | c | p{.35\textwidth} | p{.25\textwidth} | p{.2\textwidth} |}
\hline
% & \begin{sideways} RS \end{sideways} 
\textbf{Approach}
 & \textbf{RS}  
 & \textbf{P/R}
 & \textbf{Service impact}
 & \textbf{Implementation/Enforcement difficulties} 
 & \textbf{Dependencies}  \\ \hline \hline
 
 Filtering in the messaging server & CP & R & Platform components may never receive an expected message. & Unless deployed in distributed mode, can become a bottleneck. & Messaging server\\ \hline
 Filtering in each component & CP & R & Platform components may never receive an expected message. & All components should be modified to support it. & Platform components\\ \hline
 Disabling services & CP & R & Healthy components can become inaccessible. 
 
 Losing control over instances managed by disabled components. & - & Platform interfaces \\ \hline
 Replicating services & CP & P & Services should be replicated based on requirement and performance analysis of the environment. & - & Platform components\\ \hline
 Disinfecting infected components & CP & R & Healthy components can become inaccessible. 
 
 Losing control over instances managed by disabled components. & Configuration management tools and cloud platform interfaces should be deployed and configured. & Configuration management tools, Platform interfaces\\ \hline \hline
 Removing instances from the project VLAN & CP CC & R & The instance won't be accessible for the consumer and its services. & Highly dependent on the OpenStack VLANManager networking mode. & Platform components\\ \hline
 Disabling live migration & CP & R & Consumer experiences lower QoS.  & - & Platform interfaces\\ \hline
 Quarantining instances & CP CC & R & Quarantined instances won't be accessible for the consumer. & Implementing this solution requires a lot of effort as discussed briefly in \cite{aryanthesis}. & - \\ \hline
 Disinfecting an instance & CP CC & R & - & A framework for analyzing VM images and disinfecting running instances must be developed & Platform interfaces\\ \hline 
 Migrating instances & CP CC & R & Consumer may experience lower QoS. & The cloud environment should consist of distributed and independent zones. & Platform interfaces\\ \hline \hline
 Component authentication & CP & P & Small overhead for all communications. & Developing a system for managing components certificates and identity. & Messaging server and identity services \\ \hline 
 No new worker policy & CP & P & - & Developing a policy manager component & Messaging server and policy manager\\ \hline 
 Trust levels and timeouts & CP CC & P & Lower QoS for non-critical use-cases
 
  Lower resource volume for critical use-cases & High complexity & Platform interfaces, and scheduler component\\ \hline 

\end{tabular}
\caption{Comparison (RS: Responsible stakeholder, CP: Cloud Provider, CC: Cloud Consumer, P: Proactive, R: Reactive)}
\label{table:comparison}
%\end{table}
\end{sidewaystable}



\section{Conclusion}\label{section:conclusion}
% We have presented an approach to handling compromised components in an
% OpenStack IaaS configuration. Cloud Computing present some unique
% challenges to incident handling, but our experience shows that with
% proper adaptation, traditional incident management approaches can
% also be employed in a Cloud Computing environment.
Cloud computing is a new computing model. Its definitions and
realizations have new characteristics compared to other computing
models. New characteristics hinder the application 
%process 
of existing
mechanisms. In some cases, existing approaches are not applicable, and
in other cases adaptation is required. Initially, we studied different
aspects of a real cloud environment,
% We have been 
working on a
deployed environment instead of focusing on an imaginary computing
model. Experimenting on a deployed environment is helpful in reducing
the gap between academic research and industrial
deployment/requirements. 
%We should understand that 
Many questions that
are discussed in an academic environment are already solved in
industry, or are not the right questions at all. A good blog post on
this issue can be found in \cite{welsh:cloud-research}.

Although our lab setup was not big enough to be industry realistic, it
was useful for understanding the ecosystem of the cloud model, and
observing possible weaknesses in it. Obviously, deploying a larger
infrastructure reveals more information about the exact behavior of
the environment, and the result will be more accurate. However, that
may not be feasible as a university project unless big players in the
cloud are willing to contribute, as can be seen in 
% Some of those 
efforts 
%are as follows:
such as OpenCirrus \cite{opencirrus} (supported by HP, Intel, and Yahoo!),
the Google Exacycle \cite{google:Exacycle} program, and Amazon grants for
educators, researchers and students \cite{aws:grants}.

In our study we have decided to use the OpenStack cloud software. There were multiple reasons behind this decision, such as:
\begin{itemize}
\item Working on an open source project helps the community, and pushes the open source paradigm forward.
\item Analysis of the platform and experimenting with different approaches is easier and more efficient when we can access the source code.
\item Big companies are involved in the OpenStack project, and many of them are using the platform in their own infrastructure. Thus, OpenStack can become a leading open source cloud platform in the near future.
\end{itemize}
%When we started our study, it was 
Thus study was started only 4 months after the first release of OpenStack, and much of the required  documentation was either not available or not good enough. 
% were not good enough even if they were available. 
We studied the OpenStack components and identified their functionalities and other specifications. Moreover, working with a platform which is under heavy development, has its own challenges.

In order to secure the environment against a compromised component, we have to handle the corresponding incident. The NIST incident handling guideline has been studied and applied to our experimental cloud environment. During the application process we did not limit ourselves to the lab setup, because it was not large/distributed enough. So, in the proposed approaches we considered a large scale, highly distributed target environment; and made those approaches compatible with such an environment.
Moreover, the NIST guideline recommends a set of actions for each handling phase. These actions can be realized using a variety of mechanisms. We have studied several mechanisms and discussed their compatibilities with the cloud model. Additionally, we have proposed new approaches that are helpful in fulfilling incident handling requirements.
Furthermore, in this process multiple questions and challenges were raised that can be interesting topics for future work in cloud incident handling and in general security of a cloud environment. We itemize a few of them in the following:
\begin{itemize}
\item Statistical measurement and analysis of each approach and study of the exact performance overhead.
\item Large scale deployment of OpenStack with its latest release.
\item Implementation of proposed approaches as a set of security services, and study their effectiveness for a cloud consumer and the cloud environment in general.
\item Study the compatibility of approaches and guidelines to other cloud environments, specifically with those operated by industry or commercial cloud providers (e.g. Amazon, Rackspace, Google App Engine, Azure).
\end{itemize}

\section*{Authors' Contributions}
ATM performed the configuration and testing of the OpenStack lab environment, and drafted the paper. MGJ supervised the practical work, and contributed to the writing to improve the quality of the text. All authors read and approved the final document.

\section*{Competing Interests}

The authors declare that they have no competing interests.

% conference papers do not normally have an appendix
% use section* for acknowledgement
\section*{Acknowledgment}
This article is based on results from MSc Thesis work performed at NTNU. 
%% The authors would like to thank...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% The Bibliography %%
%% %% 
%% Bmc_article.bst will be used to %%
%% create a .BBL file for submission, which includes %%
%% XML structured for BMC. %%
%% %%
%% %%
%% Note that the displayed Bibliography will not %% 
%% necessarily be rendered by Latex exactly as specified %%
%% in the online Instructions for Authors. %% 
%% %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
{\ifthenelse{\boolean{publ}}{\footnotesize}{\small}
\bibliographystyle{bmc_article} % Style BST file
\bibliography{bmc_article} } % Bibliography file (usually '*.bib' ) 
%%%%%%%%%%%
\ifthenelse{\boolean{publ}}{\end{multicols}}{}


%\newpage
%\tableofcontents
\end{bmcformat}
\end{document}
